[DataFrame] Apply() for Lists and Dicts by osalpekar · Pull Request #1973 · ray-project/ray

osalpekar · 2018-05-01T08:08:09Z

No description provided.

p-yang · 2018-05-01T08:34:17Z

python/ray/dataframe/dataframe.py

You can auto-unbox this by replacing tup with (p, iwp), key or something like that.

p-yang · 2018-05-01T08:36:17Z

python/ray/dataframe/dataframe.py

Do we define behavior with non-unique columns? That will cause self._col_metadata[key] to result in a DataFrame and not a Series.

p-yang · 2018-05-01T08:36:49Z

python/ray/dataframe/dataframe.py

Same as above.

p-yang · 2018-05-01T08:45:33Z

python/ray/dataframe/index_metadata.py

You shouldn't need to change this in _IndexMetadataBase, it should be implemented in _IndexMetadata. Either way, _IndexMetadataBase is about to be deprecated by #1965

p-yang

LGTM but let's wait for #1965 to get merged in to rebase and change the constructor.

p-yang · 2018-05-01T21:30:50Z

python/ray/dataframe/dataframe.py

Add col_metadata=self._col_metadata after #1965

ping @osalpekar. Rebase and make this change, then looks great!

rebased and made change in commit below!

AmplabJenkins · 2018-05-01T21:36:35Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5132/
Test FAILed.

AmplabJenkins · 2018-05-03T07:54:43Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5164/
Test PASSed.

AmplabJenkins · 2018-05-04T05:41:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5199/
Test FAILed.

AmplabJenkins · 2018-05-04T05:43:31Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5197/
Test PASSed.

devin-petersohn · 2018-05-04T06:50:59Z

Jenkins, retest this please.

AmplabJenkins · 2018-05-04T07:22:33Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5206/
Test FAILed.

devin-petersohn · 2018-05-04T15:49:44Z

Jenkins, retest this please.

AmplabJenkins · 2018-05-04T16:57:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5216/
Test PASSed.

devin-petersohn · 2018-05-04T17:05:15Z

Merged, thanks @osalpekar

* master: (21 commits) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973) Clean up syntax for supported Python versions. (ray-project#1963) [DataFrame] Implements mode, to_datetime, and get_dummies (ray-project#1956) [DataFrame] Fix dtypes (ray-project#1930) keep_dims -> keepdims (ray-project#1980) add pthread linking (ray-project#1986) [DataFrame] Add layer of abstraction to allow OID instantiation (ray-project#1984) ...

* master: (25 commits) [DataFrame] Add direct pandas imports for MVP (ray-project#1960) Make ActorHandles pickleable, also make proper ActorHandle and ActorC… (ray-project#2007) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973) Clean up syntax for supported Python versions. (ray-project#1963) [DataFrame] Implements mode, to_datetime, and get_dummies (ray-project#1956) [DataFrame] Fix dtypes (ray-project#1930) keep_dims -> keepdims (ray-project#1980) ...

* master: [DataFrame] Add direct pandas imports for MVP (ray-project#1960) Make ActorHandles pickleable, also make proper ActorHandle and ActorC… (ray-project#2007) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973)

p-yang reviewed May 1, 2018

View reviewed changes

p-yang approved these changes May 1, 2018

View reviewed changes

osalpekar force-pushed the groupby_merge branch from ff2b9f5 to 75242b4 Compare May 3, 2018 06:47

osalpekar added 16 commits May 3, 2018 21:32

working for non-string functions and not lists of functions

4ad5d05

works with functions as strings now as well

3a8c30a

fixed linting errors

9d4739b

throwing a warning if the input is a dictionary

4913a26

added dict of lists functionality

a845336

fix minor indexing errors and lint

5e45743

removed some commented out code

e39dcf5

some comments and thoughts for apply

94afdf4

cleaned up code a little bit and added todos

acdb7a2

improved performance

317abd8

error checking and code cleanup and comments

2796551

small change

71f0869

improved list performance a lot

5402c5c

agg calls apply for lists

e9f1708

addressing comments on the PR

ffaeee4

col_metadata change

90c2b48

osalpekar force-pushed the groupby_merge branch from 75242b4 to 90c2b48 Compare May 4, 2018 04:34

updated tests to expect TypeError where appropriate

6025ae7

devin-petersohn approved these changes May 4, 2018

View reviewed changes

devin-petersohn merged commit a1d7bb3 into ray-project:master May 4, 2018

Conversation

osalpekar commented May 1, 2018

Uh oh!

p-yang May 1, 2018

Choose a reason for hiding this comment

Uh oh!

p-yang May 1, 2018

Choose a reason for hiding this comment

Uh oh!

p-yang May 1, 2018

Choose a reason for hiding this comment

Uh oh!

p-yang May 1, 2018

Choose a reason for hiding this comment

Uh oh!

p-yang left a comment

Choose a reason for hiding this comment

Uh oh!

p-yang May 1, 2018

Choose a reason for hiding this comment

Uh oh!

devin-petersohn May 3, 2018

Choose a reason for hiding this comment

Uh oh!

osalpekar May 3, 2018

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 1, 2018

Uh oh!

AmplabJenkins commented May 3, 2018

Uh oh!

AmplabJenkins commented May 4, 2018

Uh oh!

AmplabJenkins commented May 4, 2018

Uh oh!

devin-petersohn commented May 4, 2018

Uh oh!

AmplabJenkins commented May 4, 2018

Uh oh!

devin-petersohn commented May 4, 2018

Uh oh!

AmplabJenkins commented May 4, 2018

Uh oh!

devin-petersohn commented May 4, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants